{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建环境"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里的难点是，木棒会向左或向右倒，你需要在每一步，通过让\n",
    "平台往左或往右移动来保持平衡。\n",
    "这个环境的观察是4个浮点数，包含了木棒质点的x坐标、速度、\n",
    "与平台的角度以及角速度的信息。当然，通过应用一些数学和物理知\n",
    "识，将这些数字转换为动作来平衡木棒并不复杂，但问题是如何在不\n",
    "知道这些数字的确切含义、只知道奖励的情况下，学会平衡该系统？\n",
    "这个环境每执行一步，奖励都是1。片段会一直持续，直到木棒掉落为\n",
    "止，因此为了获得更多的累积奖励，我们需要以某种避免木棒掉落的\n",
    "方式平衡平台。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.04831111, -0.02519628,  0.03842328, -0.00386438])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "先重置一下环境并获得第一个观察（在新创建环境时，总\n",
    "会重置一下它）。\n",
    "\"\"\"\n",
    "e=gym.make(\"CartPole-v0\")\n",
    "obs=e.reset()\n",
    "obs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Discrete(2)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 环境的动作空间包含0，1代表只往左或者往右\n",
    "e.action_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为什么是 Box 而不是 Discrete？<br>\n",
    "离散空间（Discrete）：适用于有限、互斥的选项（如上下左右 4 个方向）。<br>\n",
    "连续空间（Box）：适用于取值范围连续的物理量（如位置、速度、角度等）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Box(4,)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "小车位置（Cart Position）：水平坐标，范围 [-4.8, 4.8]。\n",
    "小车速度（Cart Velocity）：水平移动速度，范围 [-∞, ∞]（实际受物理限制）。\n",
    "杆子角度（Pole Angle）：偏离竖直方向的角度，范围 [-0.418 rad, 0.418 rad]（约 ±24°）。\n",
    "杆子角速度（Pole Angular Velocity）：角度的变化速率，范围 [-∞, ∞]。\n",
    "\"\"\"\n",
    "e.observation_space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，通过执行动作0可以将平台推向左边，然后会获得包含4个\n",
    "元素的元组：<br>\n",
    "1. 一个新的观察，即包含4个数字的新向量。<br>\n",
    "2. 值为1.0的奖励。<br>\n",
    "3. done的标记为False，表示片段还没有结束，目前的状态多少还是\n",
    "可以的。<br>\n",
    "4. 环境的额外信息，在这里是一个空的字典。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 0.04780719, -0.22084761,  0.038346  ,  0.30068967]), 1.0, False, {})"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.step(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "动作空间样本和观察环境样本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "[-3.8788722e+00 -2.1823312e+37  1.2710987e-03 -3.3529184e+38]\n",
      "[-3.9643934e+00 -3.1258440e+38  1.7860512e-01 -1.4464549e+38]\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "动作空间有往左和往右，所以只有0，1\n",
    "观察空间则有4个随机数\n",
    "\"\"\"\n",
    "print(e.action_space.sample())\n",
    "print(e.action_space.sample())\n",
    "print(e.observation_space.sample())\n",
    "print(e.observation_space.sample())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 随机CartPole智能体"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们先创建了环境并初始化了步数计数器和奖励累积器。最后一\n",
    "行，重置了环境，并获得第一个观察（我们不会用到它，因为智能体\n",
    "是随机的）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "总奖励43.0,一共走了43步\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "if __name__==\"__main__\":\n",
    "    env=gym.make(\"CartPole-v0\")\n",
    "    total_reward=0.0\n",
    "    total_steps=0\n",
    "    obs=env.reset()\n",
    "    while True:\n",
    "        action=env.action_space.sample()\n",
    "        obs,reward,done,_=env.step(action)\n",
    "        total_reward+=reward\n",
    "        total_steps+=1\n",
    "        if done:\n",
    "            break\n",
    "    print(f\"总奖励{total_reward},一共走了{total_steps}步\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
